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CHAPTER 9 


IN-FLIGHT  ASSESSMENT  OF  WORKLOAD  USING  INSTRUMENT  SCAN 

by 

JRTole 

Digital  Analysis  Corp 
Box  2850,  Reston,  Virginia  22090,  USA 

and 

R L Harris  Sr 

NASA  Langley  Research  Center. 

Hampton,  Virginia,  USA 

INTRODUCTION 

During  instrument  flight,  the  pilot  obtains  information  concerning  aircraft  state  by  cross-checking  or  scanning  the  flight 
instruments.  The  exact  method  of  scanning  the  instrument  panel  varies  from  pilot  to  pilot  but  there  are  some  basic  features 
common  to  a “good”  scan  pattern.  Indeed,  it  was  the  early  study  by  Fitts  and  his  associates  identifying  the  most  common 
instrument  transitions  which  led  to  the  familiar  T"  arrangement  of  the  major  flight  instruments^  )jf- 

The  method  discussed  here  may  be  considered  a candidate  for  workload  studies  with  piloting  tasks  which  will  invoke  a 
regular  visual  scan  (spatial/temporal  pattern  of  eye  movements)  during  instrument  flight  When  instrument  scan  is  in  use,  it  may 
be  postulated  that  external  factors  such  as  noise,  interruptions,  fatigue,  etc  which  interfere  with  the  piloting  task  may  produce 
measurable  changes  in  the  scanning  behavior.  Such  measures  would  be  particularly  attractive  for  quantifying  workload  since 
they  would  be  both  non-invasive  and  objective. 

It  is  important  to  point  out  that  instrument  scan  by  itself  is  not  a complete  indicator  of  workload  nor  is  task  attention 
necessarily  associated  with  where  the  pilot  happens  to  be  looking  at  a particular  instant  However,  whenever  instrument  scan  is 
required  in  a piloting  task,  analysis  of  scanning  behavior  may  yield  important  direct  or  indirect  information  concerning 
workload.^. 

Scenarios  in  which  instrument  scan  may  be  considered  a potential  candidate  for  workload  assessment  include: 

1 Any  situation  in  which  instrument  flight  is  required  as  part  of  the  overall  task. 

2 Alterations  in  the  design  and/or  layout  of  cockpit  instruments, 

3 Alterations  in  controls  which  require  visual  monitoring  of. 

4 Situations  in  which  fatigue  is  suspected  to  be  high. 

METHODOLOGY 
Measuring  Visual  Scan 

Measurement  of  pilot  lookpoint  (eye  point-of-regard)  is  required  in  order  to  analyze  the  instrument  scan.  While  several 
techniques  have  been  applied  over  the  years,  the  most  practical  method  for  in-flight  measurements  is  the  remote  oculometer. 
This  device  makes  no  contact  with  the  pilot  and  does  not  restrict  his  movements  while  tracking  his  point  of  regard  to  within 
approximate  0.5  degree  accuracy.  The  oculometer  measures  infrared  light  (from  a low  intensity  source  in  a comer  of  the 
instrument  panel)  reflected  from  the  retina  and  cornea  of  the  eye  via  an  infrared  sensitive  TV  camera  and  a system  of  lenses  and 
mirrors.  Computer  analysis  of  these  reflections  is  performed  to  determine  where  the  pilot  it  looking.  Basic  output  from  the 
oculometer  consists  of  the  x,  y coordinates  of  the  visual  scene  as  a function  of  time.  Temporal  resolution  is  1/30  second.  For 
convenience  in  later  analyses,  the  raw  data  is  usually  converted  to  yield  instrument  dwell  rather  than  the  x-y  coordinates. 

NASA  Langley  Research  Center  has  devoted  considerable  effort  to  the  problem  of  installing  such  a device  in  a cockpit 
The  current  version  of  their  electro-optical  (EO)  head  requires  a little  more  than  the  space  of  an  instrument  on  the  instrument 
panel  A more  complete  description  of  the  oculometer  is  available  elsewhere  (2). 

Analyses  of  Scanning  Behavior 

Analyses  of  the  information  provided  from  the  oculometer  may  be  separated  into  temporal,  spatial  and  spatio-temporal 
categories.  In  all  cases,  the  fundamental  premise  is  that  the  ‘regular’  scan  path  will  in  some  way  be  altered  by  some  factors)  (eg 
panel  layout)  which  may  affect  workload  during  instrument  flight  The  analyses  described  here  do  not  by  themselves  measure 
workload,  however  they  allow  comparisons  of  the  scan  path  behavior  of  the  pilot  under  various  situations  and  thus  may  provide 
inferences  concerning  changes  in  workload. 

Temporal  Analyses 

Time  History  ofLookpoiru 

The  fundiunental  output  bom  the  oculometer  is  a time  history  of  lookpoint  (ie  a plot  of  the  instrument  being  viewed  as  a 
function  of  time).  Besides  providing  the  basic  data  from  which  other  analyses  may  be  performed  this  plot  is  useful  as  an 
overview  of  the  scanning  behavior,  eg  it  is  particularly  easy  to  determine  periods  of  ‘staring’  or  high  rates  of  blinking 
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Dwell  Percentages 

The  dwell  percentage  is  the  percentage  of  time  spent  looking  at  a particular  instrument.  The  transition  percentage  is  the 
percentage  of  transitions  which  occurred  between  two  instruments  regardless  of  the  direction  of  the  transition.  These  data  are 
printed  on  a schematic  view  of  the  instrument  panel  with  the  dwell  percentages  inside  the  individual  instrument  boundary  and 
lines  between  the  instruments  representing  those  transitions  which  occurred  (the  width  of  the  line  can  be  drawn  proportional  to 
the  magnitude  of  the  transition  percentage).  This  diagram  give  a graphic  picture  of  the  scan  paths. 

Dwell  Histograms 

Dwell  time  histograms  may  be  plotted  for  each  of  the  important  instruments.  Such  a histogram  is  a plot  of  the  number  of 
dwells  (looks)  on  an  instrument  which  lasted  for  the  length  of  time  indicated  by  the  abscissa.  Intuition  suggests  that  instruments 
with  either  high  information  content  or  poorer  information  transferability  will  elicit  longer  dwells  than  those  with  low  amounts 
of  information  or  good  information  transferability.  When  additional  information  is  added  to  a display  or  the  display  format  is 
changed,  dwell  histograms  may  be  successfully  used  to  examine  the  effect  of  this  change  on  the  pilot  (2).  The  goal  is  to  arrive  at  a 
display  design  which  will  provide  the  most  information  with  the  shortest  dwell  time. 

Dwell  time  histograms  tend  to  be  stereotyped  in  shape  for  different  instruments.  Dwells  can  be  classified  by  both  the 
instrument  being  looked  at  as  well  as  the  function  of  the  dwell,  ie  whether  the  pilot  was  monitoring  information  or  changing  the 
indication  by  some  control  input  while  looking  at  the  instrument.  The  histograms  for  these  two  dwell  functions  have  two  peaks, 

UlM  at  sliOTl  dwt.ll  tiliits  fui  ’thetk'  On  ail tl all  slalt  dlid  a sCtuild  ptak  at  lOngCi  d wttt  hints  asslitidltd  with  tilt  heading'  id 

aircraft  state.  The  control  dwells  show  a peak  at  a very  long  dwell  time  (2). 

Oculomotor  Dynamics 

Oculomotor  dynamics  are  a useful  type  of  ancillary  data  which  may  be  considered  during  scan  path  analyses.  While  not  a 

direct  uuitn'uuii  if  StoTlTlirig  Ovlia  ' n.  1 , tut  uttans  id  hi  W TTit  tjt  .inilts  btlWttli  itlsITiliiitiil:  may  lit  On  iilip  i lull l ii'lltitall  F if 

fatigue.  In  particular,  peak  velocity  and  acceleration  of  saccadic  eye  movements  can  be  expected  to  decease  dramatically  as  the 
oculomotor  system  fatigues.  Measurement  of  these  parameters  can  provide  an  indication  of  the  tendency  to  fatigue  under 
certain  types  of  instrument  scan. 

Spatial  Analyses 

Instrument  Transitions 

The  earliest  analyses  of  the  instrument  scan  calculated  the  probabilities  of  a pilot  making  a change  in  lookpoint  between 
pairs  of  flight  instruments.  The  instrument  transition  matrix  results  from  determining  the  probabilities  of  all  such  changes  which 
are  possible.  While  it  is  theoretically  possible  to  statistically  compare  two  such  matrices,  obtained  under  different  workload 
conditions,  the  amount  of  data  required  to  make  such  a comparison  valid  is  often  more  than  can  be  obtained  in  a practical 
situation.  This  fact  led  to  the  development  of  a single  parameter  measure  of  scan  behavior,  called  entropy,  which  in  effect 
summarizes  the  probabilities  contained  in  the  transition  matrix  (3). 

Entropy 

The  time  history  of  fixations  has  a form  which  is  similar  to  that  of  a communication  system  which  can  assume  N discrete 
states  with  a varying  duration  in  each  state.  The  orderliness  of  such  a system  is  related  to  the  probabilities  with  which  it  occupies 
its  different  states.  A system  which  always  occupied  the  same  state  or  always  made  the  same  transitions  between  states  would 
thus  be  quite  orderly.  In  the  case  of  instrument  scan,  these  situations  would  be  paralleled  by  staring  and  by  a stereotyped 
scanpath  respectively. 

This  concept  of  system  order  may  be  stated  compactly  (4)  as: 

H.  - - Z [Pilog2p,] 

i-i 


where  H0  - observed  average  entropy 

Pi  - probability  of  sequence  i occurring 
D - Number  of  different  sequences  in  the  scan 

In  the  case  of  the  instrument  scan,  entropy  has  the  units  of  bits/sequence  and  provides  a measure  of  the  randomness  (or 
orderliness)  of  the  scanpath.  The  higher  the  entropy,  the  more  disorder  is  present  in  the  scan.  The  maximum  possible  entropy  is 
constrained  by  the  experimental  conditions.  The  maximum  possible  value,  Hmax,  may  be  calculated  as  follows.  For  a given 
number  of  instruments,  M,  and  sequence  length  N,  the  maximum  number  of  different  fixation  sequences  is  given  by: 

Q — M * (M—  1)N_I  - maximum  of  sequences  of  length  N 

The  number  of  bits  required  to  uniquely  encode  all  Q possible  sequences  is  log2Q.  The  magnitude  of  tliis  latter  number 
also  represents  Hmax  of  the  visual  scan  for  the  number  of  instruments  and  sequence  length  being  considered.  For  example, 
with  7 instruments  the  value  of  Q for  sequences  of  2 instruments  is  56  which  yields  a corresponding  — 5.8. 

In  order  to  include  the  effect  of  instrument  dwell  times,  a term  for  entropy  rate  may  be  defined  as: 

H*  - Z [H/DT,] 

»-l 
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where  H,  — entropy  for  ith  sequence 

DT,  — Average  dwell  time  for  ith  sequence 
D —Number  of  different  fixation  sequences 

While  it  is  possible  for  pilots  to  make  rather  rapid  glances  (with  dwell  times  of  1 00  msec  or  less)  at  their  instruments  (5)  a 
fixation  rate  this  high  ( 1 0 fixations/sec)  rapidly  leads  to  oculomotor  fatigue.  A more  realistic  average  value  is  probably  about  2 
fixations/sec  or  less  for  a long  period  of  instrument  scan  (say  > 10  sec).  Using  this  value  (0.5  sec/look)  as  the  average  dwell 
interval,  the  maximum  entropy  rate  for  7 instruments  and  sequences  of  length  2 is  calculated  from  the  following  equation  to  be: 

(H  rate)*,,  - 5.8/0.5  • 2 fixations/seq  - 6 bits/sec 

This  number  represents  an  upper  bound.  Since  we  suspect  that  the  pilot  must  have  some  regularity  in  his  or  her  scan,  the 
numbers  we  would  expect  to  obtain  under  actual  flight  conditions  will  probably  be  lower.  The  observed  average  rate  for  the 
basic  experiments  was  on  the  order  of  1 bit/sec.  A tendency  to  stare  under  increased  load  should  be  reflected  by  decreased 
entropy  and  increased  fixation  times  making  Hrate  tend  toward  lower  values  under  such  conditions. 

Spatio-temporal  Analyses 

Comlation 

In  situations  in  which  a workload  inducing  stimulus  is  applied  either  periodically  (eg  verbal  loading,  secondary  task,  etc) 
or  in  a recurring  but  random  fashion,  the  use  of  correlation  methods  may  be  in  order. 

Autocorrelation  may  be  performed  on  scanning  data  as  follows.  A sequence  of  instrument  numbers  versus  time  is 
developed  from  the  data.  Due  to  the  arbitrary  nature  of  the  assignment  of  instrument  numbers,  the  autocorrelation  of  the  signal 
containing  all  instrument  numbers  does  not  necessarily  produce  meaningful  results.  For  this  reason  analysis  of  each  instrument 
is  examined  successively  by  replacing  the  time  sequence  of  all  instruments  with  a sequence  [x(i)|  where  the  value  is  1 for  the 
instrument  being  studied  and  0 for  all  other  instruments.  In  order  to  eliminate  the  dc  component  for  later  spectral  analysis,  a 
zero-mean  sequence  [f(i)|  is  computed  from  [x(i)|  as  follows: 

where 

**(>)-» 

if  specified  instrument  j is  being  fixated  and  0 otherwise 

Xj  - mean  of  [Xj(i)] 

The  sample  autocorrelation  of  (fj(i)|,  or  sample  autocovariance  of  [Xj(i)],  is  calculated  by  the  formula: 

Rj(k)  - I/n  Z [fj(i)*$(i  + k)] 

i- 1 


where  R,(k)  — autocorrelation  sequence  for  instrument  j 

n — number  of  samples  — total  run  duration/oculometer  sampling  period  (l/30th  sec) 

This  autocorrelation  is  computed  tor  each  instrument  for  each  loading  case.  In  order  to  detect  possible  periodicity  in  the 
scan,  the  Fourier  transform  of  the  autocorrelation  is  taken  to  produce  the  power  density  spectrum.  From  this  a value  for  the 
dominant  frequency  may  be  obtained.  For  skilled  pilots,  this  frequency  tends  to  be  close  to  that  of  the  workload  stimulus  which 
has  been  applied.  This  suggests  that  the  pilot  has  a tendency  to  multiplex  tile  flying  task  and  the  periodic  task  for  greater 
efficiency.  Overload  occurs  when  numbers  are  presented  too  rapidly  for  the  pilot  to  efficiently  multiplex  both  tasks. 

Novice  pilots,  however,  do  not  seem  to  have  any  consistent  pattern  in  their  autocorrelation  sequences.  Most  of  these  pilots 
show  little  or  no  periodicity  in  their  scans  for  any  of  the  loading  conditions.  One  explanation  may  be  that  skilled  pilots  have  a 
better  developed  ability  to  time  multiplex  several  simultaneous  tasks. 

For  stimuli  which  occur  repetitively,  but  not  at  periodic  intervals,  it  is  plausible  to  consider  the  use  of  cross  correlation 
between  the  time  at  which  the  stimuli  are  applied  and  the  scanpath  although  this  has  not  been  attempted  to  date. 

Visual  Scanning  Measures  Applied  to  the  Standard  Flight  Task,  (Manually  Flown  ILS  Approach  and  Landing  of  Two-Pilot 
Passenger  Jet  Transport). 

We  now  briefly  discuss  the  application  of  our  techniques  to  the  valuation  of  workload  during  an  ILS  approach.  Two  or 
three  factors  must  be  manipulated  to  use  the  techniques  described  above:  (a)  a piloting  task  requiring  a stereotyped  scan  path, 
(b)  a verbally  presented  mental  loading  task,  or  (c;  a visually  presented  mental  loading  ‘ask.  It  is  assumed  that  the  cockpit  to  be 
used  for  the  experiments  may  be  outfitted  with  the  NASA  Langley  oculometer  system  or  an  equivalent  and  that  ample  time  will 
be  allowed  (approximately  5-10  minutes)  for  calibration  of  the  oculometer  before  an  experimental  session  begins. 

The  proposed  ILS  approach  scenario  requires  the  use  of  a stereotyped  scanpath,  though  it  should  be  emphasized  that  the 
task  and  hence  the  scan  pattern  is  not  constant  throughout  the  scenario.  Thus,  the  second  to  second  level  of  loading  due  to  the 
flight  task  and  the  corresponding  instrument  scan  will  vary,  albeit  in  a somewhat  predictable  fashion.  The  additional  verbal  or 
visual  loading  task  serves  to  “bias”  the  total  amount  of  mental  load  on  the  pilot  with  the  goal  of  locating  peaks  in  the  load  due  to 
the  piloting  task  alone.  The  notion  here  is  that  the  workload  due  to  the  additional  task  is  roughly  additive  with  the  instantaneous 
load  due  to  the  piloting  task.  The  hope  would  be  to  bias  the  total  load  to  a high  enough  level  to  demonstrate  a performance 
decrement  (which  may  be  a non-linear  function  of  loading)  while  at  the  same  time  hopefully  observing  a monotonic  change  in 
the  measures  of  scanning  behavior  as  a function  of  the  increased  load. 
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Several  levels  of  difficulty  of  the  additional  task  are  required.  These  may  be  achieved  in  two  ways.  A constant  level  of 
difficulty  may  be  imposed  over  the  entire  approach;  this  method  is  to  be  recommended  at  present  as  we  are  not  as  yet  sure  how 
to  analyse  short  segments  of  the  scan  pattern.  Each  level  of  difficulty  of  the  imposed  extra  task  would  thus  require  a separate 
run.  Since  both  the  verbal  and  visual  tasks  are  periodic,  their  respective  difficulties  may  be  altered  during  a run  by  changing  the 
period  between  presentations  of  the  task.  This  method  would  seem  more  attractive  if  the  piloting  task  were  indeed  fixed  over 
the  entire  run. 

A verbal  task  may  be  used  as  one  means  of  biasing  the  loading  level.  This  has  been  shown  to  work  well  in  our  experiments 
and  is  easy  to  implement  and  score  (6).  Such  a task  should  be  designed  to  approximate  one  which  would  ordinarily  be 
performed  in  the  course  of  flight;  eg  a constant  rate  of  radio  communication  or  periodic  manual  computation  of  navigational 
coordinates. 

An  alternate,  visual  version  of  this  task  is  also  possible  and  perhaps  more  appropriate  for  actual  flight  conditions.  A small 
display  could  be  mounted  in  a convenient  point  in  the  pilot's  visual  field.  The  display  could  present  either  a “+"  or  a ” sign.  At 

periodic  intervals  an  auditory  “beep"  would  signal  that  the  pilot  should  observe  this  display  and  indicate  (operationally)  via  a 
rocker  switch  whether  the  display  is  currently  indicating  + or  — . The  interval  between  “beep”  determines  the  difficulty  of  this 
task  and  one  possible  measure  of  workload  is  the  % of  time  the  pilot  is  actually  able  to  observe  the  display. 

Entropy  rate  calculations  could  be  made  on  the  scanning  data  regardless  of  whether  the  visual  or  verbal  loading  task  is 
used.  Since  both  tasks  are  periodic,  the  autocorrelation  technique  may  also  be  applied.  Although  we  have  not  done  it  as  yet,  we 
expect  that  cross  correlating  the  time  of  presentation  of  the  imposed  task  with  the  scanning  data  is  likely  to  yield  good  results 
especially  in  the  type  of  flight  scenario  proposed  in  this  study.  We  expect  that  a characteristic  “signature"  will  appear  in  the  cross 
correlation  between  the  loading  task  and  the  instrument  scan  and  that  this  signature  will  be  altered  via  changes  in  task  difficulty. 

Limitations  and  Pitfalls  of  the  Technique 

There  are  a number  of  potential  problems  in  applying  our  techniques.  These  are  enumerated  below: 

1 The  piloting  task  being  performed  must  require  instrument  scan. 

2 The  relationship  between  where  the  pilot  is  looking  and  the  'focus'  of  his  attention  may  be  misleading  (clearly  this  is  the 
case  if  the  pilot  is  staring). 

3 The  scan  must  be  repetitive,  at  present,  although  it  may  be  possible  (eg  using  cross  correlation)  to  analyze  short 
segments  of  a scan  pattern. 

4 An  onboard  oculometer  is  required  and  must  be  mounted  in  the  instrument  panel  (NASA  — Langley  Research  Center 
has  worked  out  many  of  the  technical  problems  however).  Jet  Transport  simulators  at  NASA  Langley  and  elsewhere  have 
also  been  fitted  with  the  oculometer. 

5 It  may  be  necessary  to  calibrate  without  the  pilot’s  cooperation  due  to  time  limitations  in  the  proposed  experiments. 
However  sufficient  setup  time  prior  to  the  experiment  will  minimize  the  calibration  needed. 

6 The  behavior  of  the  various  measures  of  scan  has  not  been  examined  under  a wide  variety  of  situations  as  yet,  hence  we 
are  unable  to  comment  on  flight  scenarios  in  which  the  task  is  most  applicable  other  than  the  obvious  requirement  of  some 
type  of  scanning  behavior. 
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